{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 🧠 Why we need to identify `target_layers` in LoRA/QLoRA ❓\n",
    "\n",
    "📌 First, a neural network containing many dense layers performing matrix multiplication - the weight matrices in these layers typically have full-rank. Meaning that a matrix does not have any linearly dependent (i.e., “redundant”) rows or columns. In contrast, to full rank, low rank means that the matrix has redundant rows or columns.\n",
    "\n",
    "📌 The rank of a matrix in linear algebra measures the dimension of the vector space generated (or spanned) by its columns (or rows). In simpler terms, it tells us the maximum number of linearly independent column vectors (i.e. Column Rank) or row vectors (Row Rank) in the matrix.\n",
    "\n",
    "- It indicates how much information is packed into the matrix.\n",
    "    \n",
    "- For instance, if a matrix is used to represent a set of linear equations, its rank reveals the number of distinct equations.\n",
    "\n",
    "\n",
    "![](assets/2023-10-07-19-09-45.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "📌 However, when adapting to a specific task, the LORA paper shows that the pre-trained language models have a low “intrinsic dimension”.\n",
    "\n",
    "📌 A low \"intrinsic dimension\" means the data can be effectively represented or approximated by a lower-dimensional space while retaining most of its essential information or structure. i.e. we can decompose the new weight matrix for the adapted task into lower-dimensional (smaller) matrices without losing too much important information.\n",
    "\n",
    "And per LoRA Paper, the updates to the weights also have a low “intrinsic rank” during adaptation. \n",
    "\n",
    "📌 This means that they can be decomposed into the product of two smaller matrices, which significantly reduces the number of parameters required to represent the original matrix. LORA focuses on approximating these weight matrices by their low-rank counterparts.\n",
    "\n",
    "📌 Layer-wise adaptation: LORA adapts the rank of the low-rank approximation on a per-layer basis. This is based on the observation that different layers in a Transformer may have different optimal ranks.\n",
    "\n",
    "📌 So the `target_layers` are the layers that contribute the most to the model’s size and computational cost, such as the query and value projection matrices in the multi-head self-attention mechanism. 🏋️ This is due to the matrix multiplication operations required for their computation and the subsequent operations in the attention mechanism 🏋️\n",
    "\n",
    "📌 For each target layer, a low-rank approximation is computed with methods, such as singular value decomposition (SVD) or other matrix factorization techniques. The objective is to find two smaller matrices whose product closely approximates the original weight matrix in the target layer.\n",
    "\n",
    "\n",
    "```py\n",
    "from peft import LoraConfig\n",
    "\n",
    "config = LoraConfig(\n",
    "    target_modules=[\"seq.0\", \"seq.2\"],\n",
    "    modules_to_save=[\"seq.4\"],\n",
    ")\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----------\n",
    "\n",
    "## What target layers could be potential targets for LoRA 🚀\n",
    "\n",
    "Lets take a simple MLP example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch import nn\n",
    "\n",
    "class MLP(nn.Module):\n",
    "    def __init__(self, num_units_hidden=2000):\n",
    "        super().__init__()\n",
    "        self.seq = nn.Sequential(\n",
    "            nn.Linear(20, num_units_hidden),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(num_units_hidden, num_units_hidden),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(num_units_hidden, 2),\n",
    "            nn.LogSoftmax(dim=-1),\n",
    "        )\n",
    "\n",
    "    def forward(self, X):\n",
    "        return self.seq(X)\n",
    "\n",
    "# There are a few linear layers in this model that could be\n",
    "# tuned with LoRA When working with common 🤗 Transformers\n",
    "# models, PEFT will know which layers to apply LoRA to, but in this case,\n",
    "# it is up to us as a user to choose the layers.\n",
    "# To determine the names of the layers to tune:\n",
    "\n",
    "print([(n, type(m)) for n, m in MLP().named_modules()])\n",
    "\n",
    "# This should print:\n",
    "\n",
    "[('', __main__.MLP),\n",
    " ('seq', torch.nn.modules.container.Sequential),\n",
    " ('seq.0', torch.nn.modules.linear.Linear),\n",
    " ('seq.1', torch.nn.modules.activation.ReLU),\n",
    " ('seq.2', torch.nn.modules.linear.Linear),\n",
    " ('seq.3', torch.nn.modules.activation.ReLU),\n",
    " ('seq.4', torch.nn.modules.linear.Linear),\n",
    " ('seq.5', torch.nn.modules.activation.LogSoftmax)]\n",
    "\n",
    "# Let’s say we want to apply LoRA to the input layer and to the hidden layer,\n",
    "# those are 'seq.0' and 'seq.2'.\n",
    "# The corresponding config would be:\n",
    "\n",
    "\n",
    "from peft import LoraConfig\n",
    "\n",
    "config = LoraConfig(\n",
    "    target_modules=[\"seq.0\", \"seq.2\"],\n",
    "    modules_to_save=[\"seq.4\"],\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 🧠 Here are some hints about how to identify what layers could be potential targets for LoRA Finetuning 🚀\n",
    "\n",
    "In the attached code above, in the MLP class example there are a few linear layers in this model that could be tuned with LoRA. When working with common 🤗 Transformers models, PEFT will know which layers to apply LoRA to, but in this case, it is up to us as a user to choose the layers. To determine the names of the layers to tune:\n",
    "\n",
    "Let’s say we want to apply LoRA to the input layer and to the hidden layer, those are 'seq.0' and 'seq.2'. Moreover, let’s assume we want to update the output layer without LoRA, that would be 'seq.4'. The corresponding config would be:\n",
    "\n",
    "```py\n",
    "from peft import LoraConfig\n",
    "\n",
    "config = LoraConfig(\n",
    "    target_modules=[\"seq.0\", \"seq.2\"],\n",
    "    modules_to_save=[\"seq.4\"],\n",
    ")\n",
    "\n",
    "```\n",
    "\n",
    "With that, we can create our PEFT model and check the fraction of parameters trained:\n",
    "\n",
    "```py\n",
    "from peft import get_peft_model\n",
    "\n",
    "model = MLP()\n",
    "\n",
    "peft_model = get_peft_model(model, config)\n",
    "\n",
    "peft_model.print_trainable_parameters()\n",
    "\n",
    "```\n",
    "\n",
    "prints \n",
    "\n",
    "\n",
    "```\n",
    "trainable params: 56,164 || all params: 4,100,164 || trainable%: 1.369798866581922\n",
    "\n",
    "```\n",
    "\n",
    "-----\n",
    "\n",
    "For normal-sized Qlora there would be very little if any difference if you target anything else than Q and V"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
